Universal Access To All Knowledge
Home Donate | Store | Blog | FAQ | Jobs | Volunteer Positions | Contact | Bios | Forums | Projects | Terms, Privacy, & Copyright
Search: Advanced Search
Anonymous User (login or join us)
Upload

Reply to this post | Go Back
View Post [edit]

Poster: Sinkevich Date: Oct 5, 2013 5:59am
Forum: web Subject: Please, check my robots.txt

Good day.
I have a forum.

I want that ia_archiver crawls the index page and pages of partitions of forum.

myforum.com
myforum.com/forum1.html
myforum.com/forum2.html
myforum.com/forum3.html
...

But I want that ia_archiver will delete all pages of topics of forum from archive.org base.

myforum.com/topic1.html
myforum.com/topic2.html
myforum.com/topic3.html
...
myforum.com/topic99999.html

Please, check my robots.txt.
Is it true?

User-agent: ia_archiver
Disallow: /topic

Terms of Use (10 Mar 2001)